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ABSTRACT 

A study examined the effects of standards on learning 
in automotive repair programs. This report describes the methods 
followed to conduct the evaluation and present its main findings. An 
analysis of covariance design was used to test the hypothesis that 
graduates of National Institute for Automotive Service Excellence 
(ASE)-cert if ied automotive repair programs score higher on 
standardized tests of knowledge of automotive repair than graduates 
of non-ASE~cert if ied programs. Groups of students from four 
automotive programs in each of two states (secondary and 
postsecondary ASE-certif ied and secondary and postsecondary 
noncertified certified programs in both Florida and Pennsylvania) 
were given a test measuring knowledge of eight areas of automotive 
repair on which ASE certifies programs. The Test of Cognitive Skills, 
Second Edition was used to adjust the dependent variable scores. In 
all, 15 secondary-level and 41 postsecondary-level students from 
noncertified and 33 secondary-level and 47 postsecondary-level 
students from certified programs were tested. Site visits were also 
made to each program. The mean test scores (of a possible score of 
90) for the secondary-level students from certified and noncertified 
programs were 57.64 and 47.27, respectively. The mean scores for the 
pos tsecondary-level students from certified and noncertified programs 
were 67.09 and 65.68, respectively. Students from the ASE-certif ied 
programs thus scored significantly higher on the standardized test. 
(MN) 
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FOREWORD 



The Center on Education and Training for Employment of The Ohio State 
University is pleased to submit this third-party, summative evaluation to the 
National Automotive Technicians Education Foundation (NATEF). This 
evaluation compared programs that were certified as meeting the standards 
established by the National Institute of Automotive Service Excellence (ASE) to very 
similar noncertified programs. 

The noncertified programs selected for the comparison group had made the initial 
steps toward obtaining ASE certification, but at the time of the data collection were 
not certified. An indication of how close some of the comparison programs were to 
certification is that one of them became certified only two months after providing 
data as a noncertified program. 

Despite the similarity of the certified and noncertified programs, the test results 
indicate that ASE standards do have a positive influence on the learning that takes 
place in automotive repair training programs. 

The study was designed by Dr. Morgan Lewis, Research Scientist from our Center, in 
cooperation with Dr. Patricia Lundquist, the manager of the NATEF skills standards 
project, and Dr. Byrl Shoemaker, consultant to NATEF. Dr. Lewis selected the 
programs for the study and prepared this report with the assistance of Mr. Weidong 
Wang, who conducted the computer analyses. The data collection for the project, 
including the development of the test of automotive repair, was directed by Mr. 
Lawrence Gill, formerly a Vocational Education Consultant/Testing Specialist with 
the Vocational Instructional Materials Laboratory of our Center. 

On behalf of all those associated with this study, I would like to express our 
appreciation to the administrators, teachers, and students of the eight programs, 
secondary and postsecondary, certified and noncertified, in Florida and 
Pennsylvania who provided the data that made the evaluation possible. Needless 
to say, without their cooperation this study could not have been conducted. 

Ray D. Ryan 
Executive Director 
Center on Education and 

Training for Employment 
College of Education 
The Ohio State University 
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Introduction 



In May 1994, a National Skills Standards Board was established by Title IV of the 
Goals 2000: Educate America Act. With the passage of this legislation, the nation 
acknowledged that if American companies are to be effective competitors in global 
markets, they need workers whose skills are as good as any in the world. 

The partners in the automotive industry who created the National Institute for 
Automotive Service Excellence (ASE), and later the National Automotive 
Technicians Education Foundation (NATEF)i, had recognized the need for highly 
skilled workers more than two decades earlier. The cooperative and voluntary 
nature of the standards established by ASE served as a model that was adopted for 
projects to develop standards for other occupations and was incorporated in the title 
establishing the National Boa^d. 

Two years prior to the passage of the legislation, the U.S. Departments of Education 
and Labor had solicited proposals to select bidders to establish standards for various 
occupational areas. NATEF was selected by the Department of Education to update 
and expand the ASE standards. One of the conditions of this funding was that a 
third-party summ.ative evaluation be conducted to determine the effectiveness of 
the standards that were developed. This is the report of the third-party evaluation 
of the standards updated by the NATEF project. 

The Center on Education and Training for Employment (CETE), a unit of the 
College of Education at The Ohio State University, conducted the third-party 
evaluation. This report describes the methods followed to conduct the evaluation 
and presents its main finding. The results indicate that students trained in 
programs certified by ASE have a better knowledge of automotive repair than 
students in similar noncertified programs. The procedures followed to produce 
these results are discussed in the following section. 



lASE certifies both programs that train technicians and the technicians, 
themselves. NATEF is the educational arm of ASE and is responsible for 
conducting the reviews that determine if programs meet the standards set by ASE. 
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Method 



Hypotheses 

These are the hypotheses that the evaluation tested: 

1. Students from ASE certified programs score higher than students from 
noncertified programs on a standardized test of knowledge of automotive 
repair. This was the primary hypothesis of the evaluation. 

2. Students from postsecondary programs score higher than students from 
secondary programs. This provided a known-group validation of the 
dependent variable. 

3. The third hypothesis was a null hypothesis: there is no significant difference 
in the scores from Florida and Pennsylvania. 

4. None of the interactions among the independent variables are statistically 
significant. 



Study Design 

A 2 by 2 by 2 analysis of covariance design was used to test these hypotheses. The 
three dimensions of the design were defined by the following independent 
variables: 

1. Certification: program had been certified as meeting ASE standards or not 

2. Level, secondary or postsecondary 

3. State vv'here program was located, Florida or Pennsylvania 

The dependent variable was the score on a test that measured knowledge of the 
eight areas of automotive repair on which ASE certifies programs. There was a 
subtest for each of these areas. The subtests were developed by selecting items from 
the test bank of the Ohio Vocational Competency Assessment, Auto Mechanics.'^ 
The subtests for Engine Repair and Engine Performance had 15 items each; the 
subtests for the other six areas had 10 items each. The highest possible score was 
thus 90 items correct. 



2This is a new, criterion-referenced test developed by the Vocational 
Instructional Materials Laboratory of CETE. The job analysis on which the test is 
based was conducted in 1992 and the items were written and tested in 1993 and 1994. 
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The Test of Cognitive Skills, Second Edition (CRM-McGraw Hill 1984) was used to 
adjust the dependent variable scores. This is an intelligence test that is primarily 
nonverbal. Using scores on this test (hereafter called TCS/2) to adjust scores on the 
dependent variable controlled for differences in the intelligence of students that 
were likely to influence their scores on the test of knowledge of automotive repair. 
By controlling for individual difference in intelligence, the analysis yielded a more 
precise estimate of the effect of certification. 

Site Visits 

In addition to testing students at the end of their programs, NATEF-recruited teams 
conducted site visits at each of the programs that cooperated in the evaluation. 
These teams were led by experienced automotive instructors who had been trained 
by NATEF. Two automotive repair technicians from the localities served by the 
programs served as the other members of each team. The site visits were conducted 
in the same way as a regular certification visit. The cooperating schools and colleges 
were not asked to complete a fuU self-evaluation prior to the visit, but they did 
assemble background information about their programs for review by the teams. 

Sample 

As in any summative evaluation, the most difficult part of the study was defining 
and selecting the comparison group that would be used to test if ASE certification 
had a significant effect on the learning that takes place in an automotive repair 
program. The group that was selected provided a rigorous test of the effects of 
certification. 

The comparison group was selected from automotive repair programs that had 
made an initial inquiry to ASE concerning certification, but had not at the time of 
sample selection completed and returned the self-evaluation forms that are the first 
step in the certification process. The study was limited to automotive repair, 
because there were inadequate numbers of auto body and truck programs from 
which to select the certified and not yet certified programs. 

Four automotive repair programs, two secondary and two postsecondary, were 
selected in both Florida and Pennsylvania. One of the programs at each level was an 
ASE-certified program and the other was not certified, but had made an initial 
inquiry about certification. 

Attempts were made to select certified and noncertified programs in each state that 
were as similar as possible. Certified programs in high schools, regional vocational 
centers, and community colleges were matched with similar noncertified programs, 
controlling, where possible, for the populations and types of geographic areas the 
programs served. 



Results 



Validity of the Dependent Variable 

As the first step in the analysis, the subtest scores for the eight areas of automotive 
repair on which ASE certifies programs were correlated with the total scores. These 
correlations are shown in Table 1. Each of the subtests correlates highly with the 
total score; the range is from r = .70 to .85. The subtests also correlate with each other 
significantly, but at a lower level; the range is r = .42 to .66. 

TABLE 1 

INTERCORRELATIONS OF SUBTESTS FOR EIGHT AREAS 
OF AUTOMOTIVE REPAIR WITH TOTAL SCORE 



TS ER EP AT MT SS BR ES AC 



Total Score (TS) 


1.00 .79 .85 


.70 


.79 


.73 


.73 


.77 


.78 


Engine Repair (ER) 


1.00 .66 


.42 


.61 


.52 


.45 


.56 


.55 


Engine Performance (EP) 


1.00 


.49 


.66 


.56 


.53 


.58 


.61 


Auton:iatic Transmission, 




1.00 


.60 


.45 


.47 


.47 


.52 


Transaxle (AT) 
















Manual Drive Train 






1.00 


.44 


.52 


.51 


.49 


and Axles (MT) 
















Suspension and 








1.00 


.49 


.53 


.54 


Steering (SS) 
















Brakes (BR) 










1.00 


.58 


.58 


Electrical Systems (ES) 












1.00 


.60 


Heating and Air 














1.00 



Conditioning (AC) 



These correlations provide evidence for the construct validity of the test. The 
intercorrelations with the total score indicate that students tend to perform similarly 
on each of the subtests. High total scores are associated with low scores on the 
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subtests. The lower intercorrelations among the subtests indicate they are 
measuring somewhat different types of knowledge. 



The TCS/2 had a modest correlation with the scores on the automotive test, r = .38, 
and slightly lower correlations with the subtests. The correlations with the subtests 
ranged from .27 to .36. Intelligence, as measured by the TC5/2, had fairly consistent 
relationships with performance on each of the subtests, but the subtests were 
obviously measuring much more than just intelligence. Scores on the TCS/2 
accounted for 7 to 14 percent of the variability in the scores on the subtests and total 
test of automotive repair. 



Test Scores by Certification, Educational 
Level, and State 

The analysis of test scores by educational level, shown in Table 2, provides a 
different type of evidence for the validity of the test. This is a known group vali- 
dation, a comparison of groups that have had different levels of experience with the 
content area that the test measures. Those with greater experience should score 
higher. Postseconctary students, as expected, scored significantly higher than 
secondary students on the test. 

Table 2 also provides the comparisons by certification that are the focus of this 
evaluation. In three of the four comparisons, the certified programs had higher 
average scores than the noncertified programs, and there is a ready explanation for 
the one exception, the postsecondary program in Florida. 

TABLE 2 

MEAN SCORES ON TEST OF AUTOMOTIVE REPAIR 
BY CERTIFICATION, LEVEL, AND STATE 



Florida Pennsylvania Total 

Mean SD N Mean SD N Mean SD N 



Secondary 




















Noncertified 


43.83 


8.35 


6 


49.56 


11.11 


9 


47.27 


10.19 


15 


Certified 


52.42 


12.92 


12 


60.62 


13.00 


21 


57.64 


13.38 


33 


Total 


49.56 


12.07 


18 


57.30 


13.31 


30 


54.40 


12.71 


48 


Postsecondary 














65.68 






Noncertified 


61.27 


8.70 


11 


67.30 


10.39 


30 


10.19 


41 


Certified 


54.25 


15.67 


12 


71.49 


6.12 


35 


67.09 


12.01 


47 


Total 


57.61 


13.04 


23 


69.55 


8.54 


65 


66.43 


9.72 


88 



ERIC 
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The noncertified postsecondary program in Florida received its certification just two 
months after its participation in the study. When this program agreed to 
participate, it had nearly completed its self-evaluation. It used the feedback from 
this study's site visit to correct the deficiencies noted, submitted its self-evaluation 
forms, had a NATEF site visit, and was certified. 

Even though the noncertified postsecondary program in Florida had higher average 
scores than the certified program, the analysis of covariance presented in Table 3 still 
found certification to be statistically significant. 

Table 3 indicates that each of the variables used in the analysis was statistically 
significant. The significantly higher test scores in Pennsylvania were contrary to 
Hypothesis 3 of the study and contributed to significant interactions that were 
contrary to Hypothesis 4. 

The higher average score for the noncertified postsecondary program in Florida 
yielded statistically significant interactions among the variables. One of the 
assumptions of the analysis of covariance is that there are no significant interactions 
(Perenson, et al. 1983). As noted above, however, the high average for the 
noncertified program could be easily explained. Consequently, the interactions were 
eliminated and the analysis run. 

The four variables used in the analysis had a multiple correlation of R = .62 with the 
automotive test scores. The square of R, .38, indicates the proportion of the 
variability in the automotive scores that could be explained by these four variables. 

Because it was necessary to disregard the assumption of no interactions among the 
independent variables to run the analysis of covariance, another statistical test was 
run which allowed interactions. This test, generalized least squares (Berenson, et al. 
1983) is appropriate because it provides an estimate of each effect, main or 
interaction, while adjusting each effect for all other relationships among the 
variables. This test thus removes from the certification effect all variability 
associated with the other variables and their interactions. 

Table 4 presents the results of the' generalized least squares analysis including all 
interactions among the independent variables that were significant at the .05 
probability level. These interactions account for much of the explained variance. 
The important point for this evaluation is that even with all explained variation 
removed, the certification variable remains significant at less than the .05 probability 
level. 
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TABLE 3 



ANALYSIS OF COVARIANCE OF AUTOMOTIVE REPAIR SCORES BY 
CERTIFICATION, LEVEL, AND STATE 
TEST OF COGNITIVE SKILLS AS COVARIATE 



Sum of Mean 
Source DF Squares Square F- Value p. > F 



Model 


4 


8163.29 


2040.82 


8.95 


0.0001 


Error 


126 


13570.45 


107.70 






Corrected Total 


130 


21733.74 








Source 












Certification 


1 


610.06 


610.06 


5.66 


0.000941 


Level 


1 


2517.41 


2517.41 


23.37 


0.0001 


State 


1 


1953.12 


1953.12 


18.13 


0.0001 


TCS/2 


1 


1700.11 


1700.11 


15.79 


0.0001 



Note: Scores on the TCS/2 were missir\g for five students. Consequently, the 
automotive repair test scores for these students could not be used in this analysis, 

1 One-tail test with the prediction that scores of students from certified programs will 
be higher than scores of students from noncertified programs. 

Site Visit Results 

Results from the site visits had a pattern similar, but not identical, to the test results. 
The programs were evaluated using the same standards as are used to determine if a 
program qualifies for certification. ASE has established 10 standards for use in 
evaluating programs each of which are rated on a five-point scale from poor (1) to 
excellent (5). 




TABLE 4 



GENERALIZED LEAST SQUARES ANALYSIS OF AUTOMOTIVE 
REPAIR SCORES BY CERTIHCATION, LEVEL, STATE, 
AND TEST 01 COGNITIVE SKILLS 



Source 


DF 


Sum of 
Squares 


Mean 
Square 


F-Value 


p. > F 


Model 


8 


10245.93 


1280.74 


13.60 


0.0001 


Error 


122 


11487.80 


94.16 






Corrected Total 












Source^ 












Certification 






Oy i.AO 


A 1 ^ 

4, ID 


n 091 Qb 


Level 




1055.10 


1055.10 


11.21 


0.0011 


State 




677.15 


677.15 


7.19 


0.0083 


Test of Cognitive 




2547.00 


2547.00 


27.05 


0.0001 


Skills 












Certification* 




577.45 


577.45 


6.13 


0.0146 


Level 












Certification* 




588.36 


588.36 


6.25 


0.0138 


State 












LeverState 




736.64 


736.64 


7.82 


0.0060 


Test of Cognitive 




781.76 


781.76 


8.30 


0.0047 



Skill*Level 



Note: Scores on the TCS/2 were missing for five students. Consequently, the 
automotive repair test scores for these students could not be used in this analysis. 

aAll variables and interactions that are significant at the .05 probability level or less 
are shown. Interactions with probabilities above .05 are not listed. 

bOne-tail test with the prediction that scores of students from certified programs will 
be higher than scores of students from noncertified programs. 
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The standards regarding the organization, administration, and pedagogy of the 
program were rated by the team leader, who had been trained by NATEF. The 
standards regarding the technical aspects of the eight areas of automotive repair 
shown in Table 1 were rated by all three team members. Table 5 shows the mean 
ratings for the 10 standards for the eight programs, and the mean test scores of 
students from these programs. 

Because of the way the noncertified programs were selected, they tended to be very 
similar to the certified programs. The ratings reflect the simalarity in the programs. 
Nevertheless, within the two states, the certified programs received higher average 
ratings than noncertified programs. The noncertified, postsecondary program in 
Pennsylvania, however, received higher ratings than the certified secondary 
programs, and the two certified secondary progr. as received higher ratings than the 
noncertified, postsecondary program in Florida. 

The rank order correlation betVv^een the means of the site visit ratings and the test 
scores is .77, which is significant at less than the .025 probability level. The major 
discrepancy is once again the noncertified, postsecondary program in Florida which 
ranked sixth in the ratings but third in the test scores. As was noted above, this 
program corrected the deficiencies identified by the site visit and became certified 
just two months after participating in this evaluation. 

TABLE 5 

MEAN SITE VISIT RATINGS AND TEST SCORES 
OF PROGRAMS PARTICIPATING IN THE EVALUATION 



Program Means 
Site Visit Test 
Ratings Scores 



Postsecondary— certified, Pennsylvania 


4.77 


71.49 


Postsecondary-noncertified, Pennsylvania 


4.74 


67.30 


Postsecondary—certified, Florida 


4.70 


54.25 


Secondary certified, Florida 


4.69 


52.42 


Secondary—certified, Pennsylvania 


4.64 


60.62 


Postsecondary-noncertified, Florida 


4.54 


61.27 


Secondary— noncertified, Florida^ 


4.32 


43.83 


Secondary— noncertified, Pennsylvania 


3.91 


49.56 



aOnly four of the eight areas certified by ASE were rated for this program. If all eight 
had been rated, it is likely this program would have had the lowest average ratings. 



r <^ 
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Discussion and Conclusion 



This evaluation provided a rigorous test of the effect of program standards on 
learning. The noncertified programs were selected to be very similar to the certified 
programs. The outcome measure of learning used items from a newly developed 
standardized test. And differences in the intelligence of the students that were 
associated with performance on the outcome measure were statistically controlled. 

The comparison, noncertified programs were so similar to the certified ones that 
one of them-the Florida, noncertified postsecondary program-changed categories 
shortly after participating in the evaluation. This program caused the only 
difference between certified and noncertified program that was not in the expected 
direction. Even with this discrepancy, however, the statistical tests of the effect of 
certification were still significant at the .05 probability level. 

The results of this analysis make a strong case that certification improves the 
learning that takes place in an automotive repair program. To provide a rigorous 
test of the effects of the standards, the noncertified programs were selected to be as 
similar to the certified programs as possible. It is very likely that if the comparison 
group had been selected from a more representative population of all noncertified 
programs, the differences between the certified and noncertified programs would 
have been larger than those found in this study. 

Since this was a summative, not formative, evaluation, it did not attempt to 
identify the ways in which certification enhances learning, but the results of the site 
visits give some clues. The most obvious way that standards can influence learning 
is by ensuring that facilities, equipment, tools, and instruction are relevant to the 
real needs of the work place. Automotive technicians serve as members of NATEF 
certification teams to add their knowledge of local practices to the review of 
programs. Noncertified programs may not have the same level of linkage with the 
labor market. 

The ASH standards also set forth clear objectives for the knowledge and skills 
students should acquire. These objectives focus instruction and may motivate 
students by communicating clearly the expectations for satisfactory performance. 
There is a large body of research that has established that expectations can influence 
learning either positively or negatively (Rosenthal and Jacobson 1968, Swann and 
Snyder 1980). It seems unlikely that noncertified programs would have the same 
degree of clarity in their objectives. 

The goal of achieving ASE technician certification may also provide motivation. 
Students in certified programs know that the instruction they are receiving meets 
national standards. They can reasonably assume that if they satisfactorily master the 
skills they are studying, they will qualify for ASE certification after they have 
acquired the necessary on-the-job experience. 
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Conclusion 



The overall conclusion of this third-party evaluation is that ASE standards have a 
positive effect on the learning that takes place in automotive repair programs. 
Students from programs certified by ASE scored ;5ignificantly higher on a 
standardized test of knowledge of automotive repair than students from similar 
noncertified programs. It is highly likely that if the comparison programs were 
selected to be more representative of all noncertified programs, the differences 
between certified and noncertified programs would be larger than those found in 
this study. 
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